Object recognition system and abnormality detection system using image processing

ABSTRACT

An object recognition system using the image processing in which an area having a unique feature is extracted from an input image of an object, the unique image is registered in a shade template memory circuit as a shade template, the input image is searched for an image similar to the shade template registered by a shade pattern matching circuit, the position of an object is determined for each template, the speed and direction of movement of the object is determined from the positional information, and the results thereof are integrated by a separation/integration circuit, thereby recognizing the whole of the moving object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application relates to U.S. applications Ser. No. 07/692,718 filedon Apr. 29, 1991 entitled "TRAFFIC FLOW MEASURING METHOD AND APPARATUS"issued as U.S. Pat. No. 5,283,573 on Feb. 1, 1994, Ser. No. 08/018,558,entitled "Traffic Flow Measuring Method and Apparatus" filed Feb. 17,1993, as a continuation of the above-identified application, and Ser.No. 07/913,929 filed on Jul. 17, 1992 entitled "IMAGE RECOGNITION METHODAND IMAGE RECOGNITION SYSTEM", assigned to the present assignee. Thecontents of these applications are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an object recognition system suitablefor grasping motions of an object or more in particular to an objectrecognition system suitable for tracking a moving vehicle or the like,on the one hand, and to a system for detecting an abnormal phenomenon ona road or the like, or more in particular to a system for processing anddetecting an image on a TV camera.

2. Description of the Related Art

To recognize the movement or the like motion of an object by use ofimage processing is considered to provide a very effective means invarious applications. For example, vehicles running on the road arerecognized to measure the information such as the number and speed ofvehicles passed, whether they are stationary or not, etc.

Conventional systems for recognizing vehicles by processing an image onthe TV camera are described in "Vehicle Recognition by DTT Method",Computer Vision, Information Processing Society of Japan, 72-5, May1992.

To detect the traffic condition is very effective in maintaining asmooth road traffic. Systems including what is called a loop-typevehicle sensor and an ultrasonic vehicle sensor have been used fordetecting the traffic condition. These systems exert ultrasonic wave ormagnetism at a ground point on the road, measure the existence of avehicle according to the change thereof, and detect the number and speedof vehicles on the basis of the time of change. These systems, however,are basically capable of determining the traffic condition only at asingle ground point and therefore are disadvantageous in measuring awide range of conditions. For this reason, a method has positively beenused recently, in which an image obtained from the TV camera isprocessed to measure the traffic condition, as described inJP-A-2-122400. According to the conventional system disclosed inJP-A-3-204783, on the other hand, a moving object is traced bycenter-of-gravity calculation of a binary-coded input shade image fromthe TV camera. Another conventional system disclosed in JP-A-62-180488concerns character recognition but not the recognition of a mobileobject. According to the last-mentioned method, a multi-valued templateis prepared and decomposed into a plurality of binary templates, so thatsimilarity between the binary template and a binary-coded input image isdetermined by pattern matching thereby to achieve character recognition.

The prior art relating to pattern matching is disclosed inJP-A-63-98070, etc.

Further, early detection of an abnormal phenomenon on the road isimportant in maintaining a smooth road traffic.

Specifically, it is necessary to detect an accident, a stationaryvehicle, a fallen object or the like at an early time and prevent thesecondary damage from being caused by such an abnormal phenomenon.Detection of an abnormal phenomenon in a tunnel is especially important.Systems applicable to such a purpose are expected to be developed moreand more.

According to the conventional image processing systems, however, onlywhat is called "the traffic flow data" including the number and speed ofvehicles is measured, but the configuration thereof lacks means todetect various abnormal phenomena. An example of such a conventionaltraffic flow measuring system is disclosed in "Architecture of TrafficFlow Measuring System Using Image Processing" in a paper for Lecture atthe 37th National Convention of Information Processing Society of Japan,6T-6, 1988.

In the "Vehicle Recognition Using DTT Method" described above, an inputimage is differentiated and binary-coded, a binary projectiondistribution along X axis (horizontal direction) of this binary image isdetermined, and only the coordinates of this projection distributionbeyond a predetermined threshold value are stored, thus determining thetrace of vehicles. This process has been conventionally employed in mostcases of measuring the number and speed of vehicles by image processing,thereby posing the problem that it is difficult to set a binary-codedthreshold value on the one hand and measurement is difficult whenvehicles are superposed one on another on the other.

According to the conventional techniques disclosed in JP-A-2-122400,JP-A-3-204783, JP-A-62-180488 and JP-A-63-98070, the number and speed ofvehicles are measured by image processing in most cases through theprocesses of differentiation of input image, binary-coding and featuremeasurement. The problem of these methods is that a binary-codedthreshold value cannot be easily set and measurement is difficult forvehicles superposed. Also, the conventional technique for binary-codingand center-of-gravity calculation of an input shade image encounters theproblem that the image contrast is reduced by the change in theenvironment or situation in which the system is installed, therebymaking it sometimes impossible to discriminate a vehicle from thebackground. The decomposition of a multi-valued template into aplurality of binary templates for pattern matching fails to recognize amoving object accurately.

As for abnormal phenomena in a tunnel, TV cameras are not actuallyinstalled at sufficiently short intervals to monitor the entire area inthe tunnel. No one can predict where an abnormal phenomenon occurs.According to the conventional traffic flow measuring functions,therefore, it is virtually impossible for the conventional traffic flowmeasuring functions alone to measure abnormal phenomena occurringoutside of the visual field of TV cameras. Another disadvantage of theconventional systems is that all abnormal phenomena cannot be graspedwith the data on traffic flow.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention, there is providedan object recognition system comprising a shade template memory circuitstoring images at specified points of an object, a shade patternmatching circuit matching patterns between the shade template and aninput image, and an integral processing section determining the positionof an object by searching for an input image analogous to the one of theshade template, defining the speed and direction of vehicle movement bythe coordinate change of the position of the object, and identifyingthat templates identical in behaviour represent the same object, whereina unique area or feature area is picked up from an image of an object,the unique image is registered as a shade template, a point of movementis determined for each template by shade pattern matching, the speed anddirection of vehicle movement are determined from information on thedetermined points of movement, and the result of the determination isintegrated to recognize the whole of moving objects.

This technique will hereinafter be referred to as "Partial CorrelationIntegration Method (PCIM)".

According to this method, even for an object of low contrast such as inthe case of a black vehicle located in a shadow environment, forexample, portions of the object high in contrast can be registered as ashade template and integrated by post-processing. As a result, a movingobject can be followed or tracked in satisfactory manner in spite of achange in brightness or superposition of objects which may occur.

According to a second aspect of the present invention, there is providedan object recognition system further comprising a template updatingsection whereby shade templates used by being extracted from an inputshaded image are sequentially updated with the movement of an object tobe detected, so that the shade templates are sequentially updated andtherefore moving objects can be followed even when the shape or the likethereof undergoes a change over a long period of time.

According to a third aspect of the present invention, there is providedan abnormality detection system, in which "traffic flow measurement","detection of stationary vehicles", "detection of abnormally-runningvehicles" and "measurement of congestion degree" are executed within thevisual field of TV cameras, and the result of these measurements isinterpolated in space and time thereby to detect an abnormal phenomenonin other than the visual field.

Further, in order to determine various abnormal phenomena accurately,the various functions mentioned above are judged integrally orsynthetically.

By way of explanation, the "traffic flow measurement" is for measuringthe speed and number of vehicles in the visual field of TV cameras, the"detection of abnormally-running vehicles" for monitoring the runningpattern of vehicles in the visual field of TV cameras, the "detection ofstationary vehicles" for detecting a vehicle stationary or out of orderor a fallen object within the visual field of TV cameras, and the"measurement of congestion degree" for stepwise measurement of thedegree of vehicle congestion. An abnormality judgement is made byconsidering these factors as a whole. The above-mentioned functions aremeasured at each ground point, and the resulting data are spatiallyinterpolated to predict an abnormality outside of the visual field of TVcameras.

Accurate detection of various abnormal phenomena is thus made possible,and any abnormal phenomenon outside of the visual field of TV camerascan also be detected.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram for explaining an embodiment of the presentinvention.

FIGS. 2A and 2B are illustrations showing an example of processing ofshade pattern matching.

FIGS. 3A and 3B are illustrations showing an example of processing ofpartial shade pattern matching.

FIG. 4 is a diagram showing traces of "time" and "ordinates" obtained bypartial shade pattern matching.

FIG. 5 is a diagram-showing traces of "time" and "abscissa" obtained bypartial shade pattern matching.

FIG. 6 is a diagram showing a model of an object image with movingobjects followed by the partial correlation integration method.

FIG. 7 is a diagram showing traces of "time" and "ordinates" obtained byprocessing according to the partial correlation integration method.

FIG. 8 is a diagram showing a search area limitation method applied infollowing a moving object.

FIG. 9 is a diagram for explaining the relation between the speed of amoving object and the range of direction in which it moves.

FIGS. 10A and 10B are diagrams for explaining the search area for eachtemplate applied in following a moving object.

FIG. 11 is a diagram for explaining a template size determination methodwith a bird's eye view of a road.

FIG. 12 is a diagram for explaining the vehicle movement at anintersection.

FIG. 13 is a diagram showing an example of a shade pattern matchingcircuit used for updating the template.

FIGS. 14A and 14B are illustrations showing the processing executed whenvehicles are running in parallel.

FIG. 15 is a diagram showing traces of "time" and "ordinates" plottedwhen vehicles are running in parallel.

FIG. 16 is a diagram for explaining a method of assuring satisfactoryprocessing even when vehicles are running in parallel.

FIG. 17 is a diagram for explaining traces of "time" and "ordinate"plotted using a method of assuring satisfactory processing even whenvehicles are running in parallel.

FIG. 18 is a diagram for explaining traces of "time" and "abscissa"plotted using a method of assuring satisfactory processing even whenvehicles are running in parallel.

FIGS. 19A and 19B are illustrations for explaining a technique forfollowing a vehicle by correlation calculation on the basis of a shadeprojection distribution.

FIGS. 20A and 20B are illustrations showing a method of following avehicle along abscissa with correlation calculation on the basis of ashade projection distribution.

FIG. 21 is a diagram for explaining another embodiment of the presentinvention.

FIGS. 22A and 22B are illustrations showing an example of the processingof shade pattern matching.

FIGS. 23A and 23B are diagrams showing the coordinates detected by shadepattern matching.

FIGS. 24A to 24D are diagrams for explaining the manner in which amoving object is followed.

FIG. 25 is a diagram showing an example of main flow in following amoving object.

FIG. 26 is a diagram showing an example of the flow of vehicle searchprocessing in a detection area with a moving vehicle followed.

FIG. 27 is a diagram showing an example of the flow of following avehicle as a moving object.

FIG. 28 is a diagram showing an example of processing for minimizing thenumber of templates.

FIG. 29 is a diagram showing another example of processing forminimizing the number of templates.

FIG. 30 is a diagram showing the processing for minimizing the number oftemplates.

FIG. 31 is a diagram showing an example of preparation of an initialtemplate image of a vehicle.

FIG. 32 is a diagram showing the condition of a vehicle displaced froman extracted area.

FIG. 33 is a diagram for explaining a vehicle set at the center of anextracted area.

FIG. 34 is a diagram for explaining the manner in which the size of atemplate image is determined in accordance with the vehicle size.

FIGS. 35A and 35B are illustrations showing a method of updating atemplate with the vehicle size changing.

FIG. 36 is a block diagram showing the configuration of an abnormalitydetection system according to an embodiment of the present invention.

FIG. 37 is a flowchart for explaining the outline of processing oftraffic flow measurement.

FIG. 38 is a diagram for explaining the outline of processing ofmonitoring abnormally-running vehicles.

FIGS. 39A, 39B and 39C are diagrams for explaining an example of dataaccumulated as obtained for monitoring abnormally-running vehicles.

FIG. 40 is a flowchart for explaining the outline of processing ofdetecting a stationary vehicle.

FIG. 41 is a diagram for explaining the outline of processing ofcongestion degree measurement.

FIG. 42 is a diagram for explaining an example of the principle ofspatial interpolation between cameras.

FIG. 43 is a diagram for explaining an example of the principle of timeinterpolation between cameras.

FIG. 44 is a block diagram showing an example of hardware configurationof an abnormal detection system according to the present invention.

FIG. 45 is a block diagram showing a specific configuration of an imageprocessing unit in FIG. 44.

FIG. 46 is a block diagram showing a specific configuration of a centraloverall decision section in FIG. 44.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment of the present invention will be described with referenceto the accompanying drawings. By way of explanation, a road isphotographed by TV camera with the description made about a technique ofrecognizing vehicles running on the road. This technique is applicablenot only to measurement of behaviour of not only vehicles but all otherobjects.

FIG. 1 diagrammatically shows a configuration of a system according tothe present invention. An image from a TV camera 1 or the like isapplied to an image memory 3 through an A/D converter 2 for convertingan analog data into a digital data. The image memory 3 is a shade imagememory of about 8 bits. A shade template memory circuit 4, on the otherhand, is for storing shade templates of a shaded image. The shade imagememory 3 and the shade template memory circuit 4 are configured to scanthe interior of the memory by an address processor 5. The shade templatememory circuit 4 is for storing a template area cut out of an imagethrough a template extracting circuit 8. The shade template memorycircuit 4 also has a plurality of shade templates for storing variousshade images. A shade pattern matching circuit 6 is for determining thematching between the image data 30 of the image memory 3 and the imagedata 40 of the shade template memory circuit 4, and for executing thenormalized correlation calculation of a normal level or less.

In normalized correlation of an input image f(x, y) against a shadetemplate T(p, q), the similarity r(u, v) of the point of input imagef(u, v) is given as ##EQU1## where p, q are x, y sizes of a gray scaleor shade template, respectively. Any method of determining a correlationmay be used to the extent that an effect similar to the above-mentionedmethod is obtained.

The operation of the image memory 3, the scanning of the shade templatememory circuit 4 and the operation of the shade pattern matching circuit6 are all controlled by a CPU (not shown). More specifically, theaddress processor 5 and the shade pattern matching circuit 6 areactivated by the CPU, the image address given in Equation (1) isgenerated at the address processor 5, a data is read from the imagememory 3 and the shade template memory circuit 4, a data is determinedas required for similarity calculation at the shade pattern matchingcircuit 6, and the similarity r(u, v) is calculated at the CPU. Acoordinate with a large similarity, once determined, is stored in atrace management table 9 for managing the "time", "abscissa" and "time","ordinate". Then, the isolation/integration processing is effected by anisolation/integration processing circuit 10 against the trace managementtable. To explain the shade pattern matching circuit 6 briefly, the datarelating to a shade template is capable of being calculated in advance,and therefore the data relating to an input image is calculated andtransmitted to the CPU at the time of calculating a similarity. The datarelating to an input image is determined by the calculation according toEquation (4) from Equation (2) through the shade pattern matchingcircuit 6 in the above-mentioned case. The same operation may be sharedby the shade template memory circuit 4 and the image memory 3. ##EQU2##

An example of shade pattern matching of an input image is shown in FIGS.2A and 2B. In the case where a vehicle 20 is displayed on an input imagef(t) at a time point t, a shade template T1 is cut out of the inputimage f(t), raster scanning is effected at the same time as patternmatching of an input image f(t+dt) at time point t+dt, and a coordinateis sought with an increased similarity r(u, v). Then, the maximum pointof similarity is determined as shown in FIG. 2B. As a result, theinstantaneous speed and direction of vehicle movement can be determinedfrom the maximum similarity point of time point t+dt and the vehicleexistence coordinate (central coordinate of template) at a time point t.

Incidentally, the horizontal axis of an image memory is referred to as Xaxis, and the vertical axis thereof as Y axis.

The advantage of the above-mentioned normalized correlation processingis that to the extent that a registered shade template and an imagepattern are similar to each other, a similarity to some degree isobtainable even when the brightness undergoes a change or an object ishidden to some degree by something (as when vehicles are superposed ontwo-dimensional display or when only a portion of a vehicle is visibledue to a shadow of a building or the like). More specifically, eventhough similar to the road surface brightness, an object can berecognized as a vehicle as long as it has a vehicular shape. Accordingto the conventional systems using binary-coding at a certain thresholdvalue, recognition of a vehicle of low contrast is very difficult. Thetechnique according to the present embodiment, on the other hand,facilitates extraction of a vehicle.

Vehicles to be recognized are of various sizes, types and colors. Theprocessing by pattern matching is accompanied by the problem of sizechange. Pattern matching techniques disclosed in U.S. application Ser.No. 07/789,682 filed Nov. 8, 1991 are applicable to the presentinvention. The contents of that application is incorporated by referenceherein. As far as the pattern matching of whole vehicles is executed,the recognition of vehicle size is required in advance, thereby makingit difficult to apply the system to vehicles passing at random. When ahuman being observes the movement of an object of considerable size,therefore, the whole of the particular object is not always followed,but the movement of a specified point thereof is followed, so thatpoints where the movements are identical in speed and direction arefinally integrated to recognize a large object. In other words, aplurality of partial templates smaller than a vehicle are prepared, anda shade pattern matching is effected for each template. The templates,for which shade patterns are matched and the resulting speeds anddirections, are judged to be similar to each other and are integrated torepresent the same vehicle independently of the size. In the case whereno moving object exists in a registered template (in the case of animage of the road surface alone), the maximum point of similarity cannotbe defined or displaced from the central coordinate of a registeredtemplate.

As will be described in detail with reference to FIG. 3, partial areassmaller than the vehicle size are set in an input image. In FIGS. 3A and3B, templates T1 and T2 are set. These partial areas are registered astemplates respectively, and matched with sequentially applied images todetermine the maximum point of matching for each template. If thiscondition is expressed as "time" along abscissa and as "ordinate" alongvertical axis as shown in FIG. 4, a trace can be determined for eachtemplate, where the unit along time axis corresponds to a samplinginterval of image input. When an image is introduced for every threeframes, for example, the unit is 3×1/30=100 ms. This management iseffected at a trace management table 9. In view of the delicatedifference in matching point with each template, the moving speed(displacement of ordinate) undergoes a change. Nevertheless,substantially similar behaviours are obtained due to the fact that thepartial templates are for the same vehicle. As a result, an integrationprocessing of these templates makes it possible to execute therecognition of a whole vehicle. An integration processing is executed atan iteration processing circuit 10. FIG. 5 is a diagram showing tracesplotted as "time" along abscissa and "X coordinate" along ordinate. Thediscrimination of straight run and lane change is made possible byrecognizing the format of the traces. FIG. 6 shows a detailed caseinvolving two passing vehicles, in which images inputted at intervals ofa certain unit time over a period from t1 to t16 are processed.Templates carry shade images of three partial areas at the lower part ofthe screen entered by the vehicle.

The processing steps will be briefly explained below.

(1) Three partial templates T1(t1) to T3(t1) for the front part of avehicle are registered in the image of time point t1, where t designatestime, and T a template.

(2) The shade pattern matching is effected for the image of time pointt2 by use of the registered templates T1(t1) to T3(t1). As aconsequence, moving traces are determined for the partial templatesT1(t1) to T3(t1).

(3) Three partial templates T1(t2) to T3(t2) are registered also for theimage of time point t2.

(4) In similar fashion, the shade pattern matching is effected for theimage of time point t3 using the templates T1(t1) to T3(t1) and T1(t2)to T3(t2). As a result, moving traces are obtained for the partialtemplates T1(t1) to T3(t1) and T1(t2) to T3(t2).

(5) The above-mentioned steps of processing are sequentially repeated.In other words, the registration of partial templates at a given timepoint and the shade pattern matching between the partial templatesregistered at the preceding time points and the present image aresequentially repeated in parallel. The traces for respective partialtemplates are thus obtained as shown in FIG. 7. Traces for the samevehicle are located close to each other in distinction from othervehicles. The interval between two groups of close traces represents thedistance between two vehicles running in the same direction, and thewidth of each trace group the vehicle length. The inclination of thetraces, on the other hand, represents the vehicle speed.

(6) The partial templates are separated and integrated. In the case ofFIG. 7, for example, the partial templates T1(t1) to T3(t1) areprocessed on the assumption that they represent the same object.

It will be seen that the vehicle behaviour recognition is easilyexecuted by using this technique.

A specific example of each process will be explained below.

1. Method of separation and integration processings

The separation and integration processings after determining the movingpoint of each template may be executed by the technique as describedbelow.

1.1 Integration processing

(1) The trace of each template is approximated by curve according to theleast square method or the like, and those partial templates havingsmall mutual distances are integrated.

(2) The knowledge about an object image (relating to the concentrationdistribution, shape, etc.) is prepared, and determination is made as towhich part of the object is represented by the shape of a giventemplate. In this way, all the templates are integrated. In the case ofa vehicle, for example, assume that a template 1 represents "the leftimage of the tail lamp", a template 2 "the right image of the taillamp", and a template 3 "the central image of the rear window". Thisknowledge permits the determination that the template 2 is positioned tothe right of the template 1, and the template 3 is arranged above them.In other words, the templates 1 to 3 are determined to represent thesame object.

The size of a template is such that initial partial templates may beprocessed always as such (in the form of a hypothetical large template)and, after integration, as a template covering a rectangular areacontaining all the integrated partial templates.

1.2 Separation processing

(1) In the case where the feature amount (for example, averageconcentration) of some of the integrated templates suddenly increases,they are separated from the integrated pattern.

(2) The templates thus integrated are assumed to be coupled under acertain resistance. In the case where the behaviour of a given templateis different from that of another template connected thereto, therefore,they are left integrated if the behaviour is not to such a degree as tocut off the resistance. If the difference is so great as to cut off theresistance, on the other hand, the affected template is separated. Theresistance may be defined as a weight representing the strength ofcoupling or a function representing the degree of coupling.

2. Method of limiting search range

In recognizing the behaviour of a vehicle running along the road, fullscanning of the correlation calculation of registered templates is notalways necessary. More specifically, in view of the fact that the rangeof movement of a vehicle is limited to some extent, the next range ofsearch can be specified by the past behaviour of the vehicle. Theprocessing time of the similarity calculation is proportional to thenumber of picture elements (the sizes of templates and search range),and therefore, the search range should be reduced as far as possible. Anexample is shown in FIG. 8. In the case where a vehicle moves from thecoordinate (x0, y0) to (x1, y1), for instance, it is possible to specifythe next search area by using the moving speed V and the direction ofmovement θ. A predicted position is included in a fan-shaped range shownin FIG. 8 as determined from the minimum speed Vmin, the maximum speedVmax and the amount of change in the direction of movement 2φ againstthe preceding moving speed V. To simplify the processing, therectangular area surrounding the fan-shaped area is used as the nextsearch range. Such a search range is determined according to Equations(5) to (8) below.

    xs=x1+Vmin·cos(θ+φ)                     (5)

    ys=y1+Vmax·sin (θ+φ)                    (6)

    xe=x1+Vmax·cos(θ-φ)                     (7)

    ye=y1+Vmin·sin (θ-φ)                    (8)

where (xs, ys) represents the starting coordinate and (xe, ye) the finalcoordinate of an area. The range φ of the vehicle direction may bechanged as a function of the speed V as shown in FIG. 9. Assume that theobject of recognition is other than a vehicle which is simple inbehaviour pattern, or, that a human being, an animal or the like objectis to be recognized and followed, for example, a simple search range asdetermined above is sometimes insufficient. In view of this, the pastbehaviour pattern is learned by a neural network in order to specify thenext mobile range from the traces of past movement. If this behaviourpattern is used for prediction of the future behaviour, the resultingsearch range which will be more detailed is expected to reduce theprocessing time.

As a result, in following the vehicle coordinates P1 to P4 determined ata time point t as shown in FIG. 10A, a search area can be determined asshown in FIG. 10B, thus remarkably reducing the processing time.

3. Method of updating templates

In spite of the fact that the foregoing description concerns the case inwhich a road is viewed from right above, it is almost impossibleactually to photograph a road from overhead. A bird's-eye view of a roadis the result. In such a case, a vehicle appears smaller with theascendance in the screen as shown in FIG. 11. In the method of followinga vehicle according to the above-mentioned correlation calculationmethod, therefore, a vehicle located at a far point cannot be detectedand therefore cannot be followed, although detection is possible ofvehicles appearing to be of the same size as the registered shadetemplates. This problem never fails to arise in applications involving awide monitor range such as in monitoring human intruders or watchingrailroad crossings.

In view of this, a shade template is updated while following themovement of vehicles as shown in FIG. 11. More specifically, in the casewhere templates with the ordinate centered at Y1 is registered from theimage of time point t1, assume that a similarity point is detected atthe ordinate Y2 by shade pattern matching of the image of time point t2.The template size in registration is reduced in accordance with theratio determined by the coordinate Y2, and the shade image at time pointt2 in the vicinity of the similarity point is registered again as atemplate. The shade template newly registered is used for matching inthe next search area. Sequential executions of this process permits theversatile operation of following vehicles against changes in vehiclesize. This method has been developed on the basis of the fact that thesize and direction of movement of a vehicle does not substantiallychange instantaneously but slowly.

A template may be prepared, instead of by the above-mentioned methodusing a latest input image, alternatively according to the average((f+g)/2) of the preceding template image (g) and an image newlydetected in the vicinity of a vehicle position (f) or a linear couplingcalculation (αf+βg, α+β=1).

Assume that the number of vehicles turning to the right at anintersection is measured by using the above-mentioned technique as shownin FIG. 12. In view of the fact that the vehicles turning to the rightchange the direction thereof gradually, the following of a vehicleceases midway with a single shade template. The following of vehicles,however, is facilitated by updating the shade template at the same time.Also, the movement of a human being constantly changing in shape iscapable of being followed by updating a template as described above.

When a shade template is updated, however, the data relating to theshade template in the calculation of Equation (1) mentioned above cannotbe determined in advance. This value is thus required to be determinedwith rapidity. In view of this, a circuit may be added as shown in FIG.13 for acquiring the data on the shade template T (i, j), so that withthe registration of a shade template, the calculations of Equations (9)and (10) are executed to always permit high-speed processing. ##EQU3##where p, q are constantly variable with the size of the shade template.

A template is extracted from an input image by a template extractingcircuit 11. The next template is prepared at a template update circuit12 by averaging or linear coupling of the template thus extracted andthe preceding template. The resulting template is registered and usedfor the next operation of following vehicles. A device for continuousfollowing operation is thus configured.

4. Size of template

Any case of vehicles running in parallel is not described above.Actually, however, a road often has two or three lanes, and therefore,it is necessary also to be able to process the recognition of vehiclesrunning in parallel. A case of recognition of the behaviour of vehiclesrunning in parallel is shown in FIGS. 14A and 14B. Templates T1 to T4are registered with respect to an input image as shown in FIG. 14A. Whenthese templates are used to follow the vehicle behaviour by shadepattern matching against an input image of time point t+dt, a matchingpoint is detected for each template. If a moving trace is plotted as"time" along abscissa and "ordinate" along vertical axis, the diagram ofFIG. 15 is obtained. It will be noted that in the case where the speedand direction of movement of two vehicles are substantially identical toeach other, mere analysis of the traces shown in FIG. 15 cannotdiscriminate small and large vehicles.

This is due to the fact that the area on a road separating the twovehicles fails to be recognized. More specifically, a templateexclusively covering the road cannot be registered, if each template isexcessively large in size. It is, therefore, important to set thetemplate width to not more than the vehicle intervals (along Xdirection). The operation with a reduced template size is shown in FIG.16. One or more templates (corresponding to T5 in the case underconsideration) of a size smaller than the interval between vehicles arealways provided and are followed with the shade pattern matchingdescribed above. The time-ordinate and time-abscissa relationships inthis state are shown in FIG. 18. In the time-ordinate graph, whichtemplates are associated with the same vehicle is unknown when twovehicles are running at substantially the same speed. From theobservation of the time-abscissa graph, however, an area lacking avehicle is detected, and therefore the templates T1 to T4 and T6 to T10are known to belong to the same vehicle.

As explained above, it is important to set the template width (along Xdirection) to not more than the interval between vehicles in order toenable a separating point to be recognized between vehicles. At the sametime, the template length (width along Y direction) may be of any sizeif in a range processable by correlation calculation (such a size as topermit grasping the features of the vehicle).

5. Method of template registration

A method of registering a template will be explained. A method ofextracting a template followed includes using an image of apredetermined area (the simplest method being by using templatesarranged on the vehicle-incoming side as shown in FIG. 16) always as atemplate. According to this method, however, a vehicle cannot befollowed if it reappears after being hidden behind a large vehicle orthe like. In view of this, a shade image of input associated with aunique point of the whole image, such as the one with a largedifferentiation value (spatial frequency feature) or the one with alarge difference between images obtained by differentiations taken atpredetermined time intervals is set as a template, and each templatethus set is followed by the correlation calculation. These templates maybe combined with images on the vehicle-incoming side.

As another alternative, a learning processing is effected in such amanner as to extract an object directly from an image by use of a neuralnetwork, and a candidate image of a moving object is directly extractedby use of the result of learning, so that a shade image of an inputcorresponding to the particular area is extracted as a template.

By sequentially executing this processing, even if a vehicle appears inthe midst of processing, the following of vehicles becomes possible atand after the time of appearance.

6. Method of correlation calculation

Although the above-mentioned "partial correlation integration method" isfor executing the two-dimensional correlation calculation, the "methodof correlation integration by concentration projection" involving asimplified processing has a similar effect. As an outline is shown inFIGS. 19A and 19B, the correlation calculation is effected on the basisof a one-dimensional concentration distribution for determining aconcentration projection (concentration accumulation) along ordinate foreach lane. If a partial pattern within a predetermined width of thisprojection distribution is followed as a template, the amount ofmovement along ordinate is determined. The change in abscissa, however,cannot be detected. If the detection along abscissa is to be executed,the projection (concentration accumulation) along abscissa of theconcentration in the vicinity of a matching point along ordinate isdetermined when the particular matching point is obtained as shown inFIGS. 20A and 20B, and the correlation with a template in registrationis defined. In this way, the change along ordinate is determined. Thegeneral procedure for the above-mentioned method is the same as the"method of partial correlation integration". Templates may be registeredby a method placing sole emphasis on the change points of the projectiondistribution. The advantage of this method lies in that the processingtime is reduced considerably as compared with the "partial correlationintegration method" described above due to the fact that the template isone-dimensional.

Although the above-mentioned methods refer only to the correlationcalculation of a shade image, the binary template matching orhierarchical pattern matching (both shade and binary) which have so farbeen employed may be executed. In such a case, too, it is important toutilize the size, separation/integration of templates or updating oftemplate patterns.

The "method of partial correlation integration" and "method of partialcorrelation integration by concentration projection" according to thepresent invention can be used for wider applications. Apart from thehuman being or vehicles as an object, these methods are applicable alsoto the operation of following other types of moving objects. Someapplications include:

(1) Detection of stationary vehicles: By following vehicle traces, theposition where a vehicle has stopped is determined. The illegal parking,etc. is thus detectable.

(2) Management of parking lot: A parking lot is monitored from a highpoint to recognize an unoccupied area by following vehicle traces.

(3) Monitoring crossings: Whether any vehicle stays within the crossingrange is recognized. The vehicle behaviour is a simple movement alonglateral direction (or along vertical direction depending on the cameraposition), and therefore vehicles can be followed in simple fashion.

(4) Monitoring intruders (security): Whether any suspicious person hasintruded during the night time, for instance, is recognized. Since ahuman being is an object of recognition, a partial template smaller thanthe width of the human being should be used. The detection of anintruder into the railroad track from the station platform can also beprocessed.

According to the present embodiment, the behaviour of an object isaccurately detectable by the partial correlation integration method evenin the case of low contrast or superposed objects. This compares withthe conventional methods in which an input image is binary-coded or athreshold value is set for a binary projection distribution, with theresult that tuning is difficult with the processing performance subjectto a considerable variation depending on such conditions as environmentsincluding brightness and shadow.

The present embodiment, which lacks binarized threshold values,eliminates the tuning, and is applicable to environmental changes inversatile fashion. Also, moving objects can be followed by shade patternmatching by the updating of shade templates even in the case where theshapes of templates stored and the moving objects to be followed undergoa change. The invention is thus easily realizable for recognition ofvehicles turning to the right at crossings or the behaviour of humanbeings. In addition, templates are traced as partial templates, andseparated and integrated in post-processing, so that objects of varyingsizes can be easily recognized.

Another embodiment of the present invention will be described below withreference to the accompanying drawings.

FIG. 21 is a block diagram showing a configuration of a vehiclerecognition system according to another embodiment of the presentinvention. The image of a road taken by a TV camera 1 or the like isapplied to an image memory 3 through an A/D converter for converting ananalog into a digital data. A shade image memory of about 8 bits (256gradations) is used as the image memory 3 according to the presentembodiment. This vehicle recognition system includes a shade templatememory circuit 4 for storing shade templates of shade images (8 bits and256 gradations) of a vehicle. The shade image memory 3 and the shadetemplate memory circuit 4 are so configured as to scan the interior ofan memory by an address processor 5. The shade template memory circuit 4has a plurality of shade templates which have stored therein shadeimages of various vehicles taken from various angles. The shade patternmatching circuit 6 is for matching between the image data 30 of theimage memory 3 and the image memory 40 of the shade template memorycircuit 4, and executes the normalized correlation calculation as shownby Equation (1). This embodiment is configured with the template cut-outcircuit 8, the trace management table 9 and the separation/integrationcircuit 10 eliminated from the system shown in FIG. 1.

Any method of correlation processing having a similar advantage may beemployed with equal effect.

The operation of the image memory 3, the scanning and shade patternmatching circuit 4 and the operation of the shade pattern matchingcircuit 6 are controlled by the CPU 7. More specifically, the addressprocessor 5 and the shade pattern matching circuit 6 are activated bythe CPU 7, an image address of Equation (1) is generated at the addressprocessor 5, the data on the related address of the shade templatememory circuit 4 and the image memory 3 is read out, the data requiredfor similarity calculation is determined at the shade pattern matchingcircuit 6, and the similarity r(u, v) is calculated at the CPU 7.

The shade pattern matching circuit 6 will be briefly explained. The datarelating to the shade template is capable of being calculated inadvance. At the time of similarity calculation, therefore, the datarelating to the input image is calculated and transmitted to the CPU 7.The data relating to the input image, in the case of Equation (1), isobtained at the shade pattern matching circuit 6 by the calculation ofEquations (2) to (4).

A diagram for explaining the shade pattern matching against an inputimage is shown in FIGS. 22A and 22B. Assume that four vehicles 13 are inthe view as an object of matching as shown in FIG. 22A. Shade templatesrepresenting the color and size (shape) of the vehicles are required tobe prepared. While scanning with the pattern matching effected with ashade template T21 as shown in FIG. 22B, for example, a coordinateassociated with an increased similarity r(u, v) is searched for. A pointP1 is determined as shown in FIG. 23A. In similar manner, the scanningwith shade templates T22 to T24 makes it possible to determine points P2to P4 as shown in FIG. 23A. As a result, once the coordinates of vehicleposition at time point t and at time point t+Δt are determined as shownin FIG. 23B, the instantaneous moving speed and direction of eachvehicle can be calculated.

The advantage of the afore-mentioned normalized correlation processingis that to the extent that the pattern and the image of shade templatesin registration are similar, a similarity is obtained to some degreeeven when brightness changes or a vehicle is hidden behind something (asin the case of vehicles superposed or hidden partially in the shadow ofa building). More specifically, even a vehicle bearing a color similarto the brightness of the road surface can be recognized as a "vehicle"from the shape thereof. In conventional systems, the binary-codingoperation at a certain threshold value makes it very difficult torecognize a vehicle of low contrast. The method according to the presentembodiment, by contrast, permits easy extraction of a vehicle.

In recognizing the behaviour of a vehicle running along the road, fullscanning for vehicle search as shown in FIG. 22B is not always required.More specifically, in view of the fact that the moving range of avehicle is limited to some degree, the next search range can bespecified from the preceding behaviour of the vehicle. The limitation ofthe search range is executed in the same way as in the embodimentdescribed above with reference to FIGS. 8 to 10A, 10B and Equations 5 to8 and will not be explained again.

Now, explanation will be made with reference to FIGS. 24A to 24D aboutan example in which a vehicle is followed actually.

FIG. 24A shows the case in which vehicles move upward in the screen.Vehicles are required to be followed sequentially by the images thereofentering the screen. For this purpose, a detection area is set at thelower part of the screen, and vehicle images entering this portion arefollowed upward. Several shade templates are prepared. First, a vehicleis detected in a search area. More specifically, the similarityprocessing is executed in the search area for each shade template.Assuming that vehicles are detected at coordinates P1 and P2 as shown inFIG. 24B at time point t as a consequence, a following table is preparedas Table 1, in which the coordinates P1 and P2 are registered at Xoldand Yold, respectively.

                                      TABLE 1                                     __________________________________________________________________________    Conditions at time t                                                                                             Next search                                flag                                                                             Xold                                                                             Yold                                                                             Tno                                                                              Timeold                                                                            x1 Y1 Time1                                                                             X2                                                                              Y2                                                                              Time2                                                                             area                                       __________________________________________________________________________    1  x0 y0 3  t    x0 y0 t   0 0 0   xs,ys,xe,ye                                1  x10                                                                              y10                                                                              5  t    x10                                                                              y10                                                                              t   0 0 0                                              .                                                                             .                                                                             __________________________________________________________________________

At the same time, the shade template number Tno and the time Timeold areregistered. Since the initial states of following are involved, Xold,Yold and Timeold are registered at X1, Y1 and Time1 respectively, and"0" at X2, Y2 and Time2. X1, Y2 and Time1 represent the precedingcoordinate and time, and X2, Y2 and Time2 represent the presentcoordinate and time for following vehicles. Further, due to the initialstages, the preceding speed and direction of movement are notdetermined. As the next search area, therefore, the initial moving speedof "0" to maximum speed (150 km/h on an expressway, for example) and theinitial change amount of the direction of movement of, say, 30 degrees,are set, with a moving range designated (xs, ys, xe, ye are calculatedwith Vminat 0 km/h, Vmax at 150 km/h, θ at zero degree and φ at 30degrees). An image at time point t1 is inputted, and the similarity iscalculated with a shade template with Tno of "3" with respect to thesearch area for vehicle P1. As shown in Table 2, the point P1' ofmaximum similarity is substituted into the coordinate (X2, Y2).

                                      TABLE 2                                     __________________________________________________________________________    Conditions at time t1                                                                                              Next search                              flag                                                                             Xold                                                                             Yold                                                                             Tno                                                                              Timeold                                                                            x1 Y1 Time1                                                                             X2 Y2 Time2                                                                             area                                     __________________________________________________________________________    1  x0 y0 3  t    x0 y0 t   x1 y1 t1  xs',ys',xe',ye'                          1  x1 y10                                                                              5  t    x10                                                                              y10                                                                              t   x11                                                                              y11                                                                              t1                                           .                                                                             .                                                                             __________________________________________________________________________

Similar processing is executed also for the vehicle at point P2. In thesecond and subsequent executions, the moving speed can be determined.The moving speed V thus determined and the direction θ are used todetermine and store the next search area by the method mentioned above.This is also the case with the processing at time point t2, which isexpressed as in Table 3.

                                      TABLE 3                                     __________________________________________________________________________    Conditions at time t2                                                                                              Next search                              flag                                                                             Xold                                                                             Yold                                                                             Tno                                                                              Timeold                                                                            x1 Y1 Time1                                                                             X2 Y2 Time2                                                                             area                                     __________________________________________________________________________    1  x0 y0 3  t    x1 y1 t1  x2 y2 t2  xs',ys',xe',ye'                          1  x1 y10                                                                              5  t    x11                                                                              y11                                                                              t1  x12                                                                              y12                                                                              t2                                           .                                                                             .                                                                             __________________________________________________________________________

This management makes it possible to execute the following of vehicleswith ease.

FIGS. 25, 26 and 27 are flowcharts showing specific steps of processing,in which FIG. 25 shows a main flow.

(1) First, shade templates are registered and the management table isinitialized.

(2) An image is inputted, and vehicles are searched for in a detectionarea with respect to the particular image, thereby executing thefollowing of vehicles.

The vehicle search in the detection area is executed as follows, asshown in FIG. 26:

(1) The shade template number is initialized, and the matchingprocessing executed by the template number i.

(2) If there is any coordinate with a similarity degree more than athreshold value, the maximum coordinate for such a similarity degree isdetermined, and the coordinate, the shade template number and the timeare registered as Xol, Yold, Tno and Timeold respectively in thefollowing (tracking) table.

(3) In the process, values of Xold, Yold and Timeold are stored in X1,Y1 and Time1, respectively.

(4) Then, a flag of the table is set and the next search area (initialvalue) is determined.

The above-mentioned processing is executed also for all the shadetemplates. The vehicle detection process is thus complete.

The processing for following vehicles is executed as shown in FIG. 27.

(1) The counter of the following table is initialized. The patternmatching is effected by the shade template (Tno) with respect to thenext table if the flag in the following table i is "0", and with respectto the search area if the flag is "1".

(2) If there is any similarity degree more than a threshold value, thecoordinate of maximum similarity degree is extracted. The coordinate isstored at X2, Y2 of the following table, the prevailing time at Time2.

(3) A search area for the next image is calculated from the moving speedand direction determined from X1, Y1, X2, Y2, and the search area isstored.

(4) After that, X2, Y2 and Time2 are copied at X1, Y1 and Time1respectively.

The afore-mentioned operation is performed for all the vehicles with aset flag, thereby completing the vehicle-following processing.

Provision of a number of shade templates for detection of incomingvehicles would consume a very long time in vehicle search at a detectionarea in FIGS. 24A to 24D, with the result that moving objects, if highin speed, could not be detected one by one. This problem may be solvedby either of the two countermeasures described below.

(1) To reduce the shade templates as far as possible.

(2) To prepare shade templates for following vehicles from an inputimage.

The method (1) will be explained. Shade templates of only "white/small","black/small", "white/large" and "black/large" used for vehicles areinsufficient in number. In the case where a white/small template isavailable, for instance, the vehicles, though all small in size, cannotbe detected by the matching with patterns of different shapes due tovariations including sedan, van, jeep and vehicle with loading space. Itis, however, very time-consuming and unrealistic to store all types ofshade templates for matching.

In view of this, a method of preparing shade templates will be explainedwith reference to FIGS. 28 to 30. Assume that there are eight shadetemplates T(1) to T(8) prepared as shown in FIG. 28. The similaritydegree of each template in comparison with the others is checked bymatching, say, template T(1) with the other shade templates. If there isany pair of similar templates, both the templates of the particular pairare used to prepare a new pattern of template, while the original one isdiscarded thereby to reduce the number of templates. In the case wherepattern T(1) is similar to pattern T(6), and pattern T(3) to patternT(5), for example, the images of both patterns T(1) and T(6) areaveraged to prepare a pattern T'(1). Also, a pattern T'(3) is preparedby averaging the images of patterns T(3) and T(5).

As a result of combining images by determining the similarity once inthe manner mentioned above, the shade templates are reduced to six asshown in FIG. 29. By similar matching between the patterns shown in FIG.29 and the initial shade templates T(1) to T(8) shown in FIG. 28, newshade templates shown in FIG. 30 are synthesized, thereby reducing thetotal number of templates. If this process is repeated until thesimilarity degree larger than a certain threshold value is eliminated,the shade templates finally obtained meets the necessary minimum forextracting vehicles. The processing time is thus remarkably reduced.Instead of preparing a new shade template by averaging images asmentioned above, the maximum or minimum concentration of two images maybe reserved.

Now, the method (2) will be explained. According to the method (2),shade templates are not prepared in advance. Instead, images to befollowed are automatically cut out of an input image taken by way of TVcamera, and are used as shade templates. In this case, as many templatesas vehicles to be followed suffices.

This method will be explained with reference to FIGS. 31 to 33. FIG. 31shows a case in which a vehicle is moving upward in the screen (thedashed line defines lane areas). In order to follow this vehicle, anarea sufficient covering the vehicle is cut out of the image at thelower part of the screen, and is used as a shade template as shown inFIG. 31. If an image not covering the vehicle at the center of the areato be cut out is registered as a shade template as shown FIG. 32, thesubsequent following process is adversely affected. The concentration ofthe image thus cut out is checked for any deviation to periphery, and ifneed be, the position and size of the area to be cut out are changed.

Taking note of the fact that the vehicle pattern is laterally symmetric,the processing mentioned below is executed.

(1) The cumulative distribution of concentration along verticaldirection of the cut-out image is determined (cumulative distribution ofconcentration along abscissa).

(2) The center of gravity xg and the variance σx of the concentration ofthe distribution thus obtained is determined.

(3) In similar fashion, the cumulative distribution of concentration isdetermined along horizontal direction of the image cut out (cumulativedistribution of concentration along ordinate).

(4) The center of gravity yg and the variance σy of concentration of thedistribution thus obtained are determined.

(5) With the concentration center of gravity xg, yg as a centralcoordinate, an area of the size given as

    Δx=α·σx

    Δy=α·σy

is cut out, where α is a constant (FIG. 33).

(6) The processing mentioned above is executed again for the cut-outarea and is repeated until the center-of-gravity coordinate and variancecome to remain unchanged.

The aforementioned processing makes it possible to register a shadetemplate of a minimum area (FIG. 34) surrounding a vehicle. In the casewhere the concentration is sided in the end, the vehicle is assumed tobe not covered completely in the screen, and the registration of theshade template is suspended (and reprocessed with an image inputted atthe next time point).

The embodiments described above refer to the processing with a roadviewed from right above. In actually taking a picture of a road,however, it is virtually impossible to photograph a road from rightabove. The inevitable result is taking a bird's-eye view of a road. Insuch a case, the road presents such a view that a vehicle, with themovement from T1 to T2 starting with time T0, appears the smaller thefarther it runs away, as shown in FIG. 35A. If a shade template isfollowed in the manner explained with reference to FIG. 31, a vehicleappearing to be of the same size as the shade template could be detectedbut a far vehicle could not be detected and finally fails to befollowed.

This inconvenience is overcome by updating the shade template whilefollowing the vehicle as shown in FIG. 35B. More specifically, when apoint P0 is detected by a shade template, another shade template for thenext search is prepared by use of an image proximate to the point P0. Inthis method, as in the method described above, the concentration centerof gravity and variance may be used. The shade template thus newlyregistered is used for matching processing for the next search areathereby to determine the point P1. A shade template is prepared by usingan image in the vicinity of the point P1. This process is sequentiallyexecuted, so that vehicles, even if changed in size, can be followed inversatile fashion. This method is based on the fact that the vehiclesize rarely changes instantaneously.

A template may alternatively be prepared by linear coupling operation(αf+βg, α +β=1) or by determining an average ((f+g)/2) of the image (f)in the vicinity of a newly-detected vehicle position and the precedingtemplate image (g).

If this technique is used, a vehicle can be followed easily by updatingthe shade template at the same time, unlike when using a single templatein which case the following of a vehicle is suspended midway inmeasuring the number of vehicles turning to the right at anintersection, in view of the fact that the vehicle turning to the rightchanges its direction slowly, as shown in FIG. 12. An alternative methodconsists in reducing the size of the shade template according to thevehicle size, i.e., according to the ratio determined by the ordinate ofthe vehicle position determined. In this method, however, the changeonly in size can be dealt with.

According to the present embodiment, as described above, the behaviourof vehicles, even if low in contrast or superposed, can be detectedaccurately by the shade pattern matching. Also, the updating of a shadetemplate makes it possible to execute the following of a moving objectby pattern matching even when the shape of the template stored or themoving object undergoes a change. Especially, easy recognition of avehicle turning to the right at an intersection or the like is possible.

This embodiment is also applicable to the detection of a stationaryvehicle (such as illegally parked) or management of a parking lot(detection of an occupied section).

Another embodiment of the present invention as applied to an abnormalitydetection system for a road, etc. will be explained.

A general configuration of the present embodiment is shown in FIG. 36.An abnormality detection system 200 includes a plurality of imageprocessing units 100-1, 100-2 and 100-3 and a central overalldetermination section 22. In order to grasp various abnormal phenomena,each of image processing units 100-1, 100-2 and 100-3 includes a trafficflow measuring section 26 for collecting various information byprocessing input images of objects, an abnormal run monitor 14, astationary vehicle detector 16, a congestion degree measuring section 18and a local overall determination section 19 (19-1, 19-2, 19-3). Thecentral overall determination section 22, on the other hand, is forcollecting data on each image processing unit and making an overalldetermination by spatial interpolation, for example.

The operation of each section will be briefly explained. The imageprocessing unit 100 is inputted with an image (section 23), extracts abackground difference feature with respect to the image (section 24),follows vehicles (section 25), and measures the resulting traffic flow(number and speed of vehicles) (section 26). Also, the frame feature ofan input image is extracted (section 15) and a stationary vehicle isdetected (section 16). In addition, features of spatial and temporaldifferentiations with respect to an input image are extracted (section17) and the degree of congestion is measured (section 18). An overalldecision is made on these results at the local overall determinationsection 19 (20, 21) to determine the presence or absence of any abnormalphenomenon. At the same time as the abnormality determination at eachunit, all the measurement information such as congestion degree andspeed obtained at each unit are collected by the central overalldetermination section 22, and after spatial interpolation, an abnormalphenomenon outside of the visual field of the camera is detected.

Each section will be described below in detail.

(1) Traffic flow measurement section

The traffic flow measurement data includes the vehicle speed and thenumber of vehicles passed. An example of the method for obtaining thesedata is shown in FIG. 37. The steps for processing are described below.

The background image 30 stored in advance is removed from the inputimage f(t) 31 at time point t thereby to extract a vehicle.

The end coordinate of the vehicle image 32 extracted (as in the casewhere the vehicle is photographed from behind) is detected (33).

The above-mentioned steps 35 and 36 are executed with the image f(t+dt)sequentially obtained, and the position to which the end coordinate hasmoved is determined by the coordinate-following operation 37.

The number of vehicles passed 38 and the speed 39 are measured from thecoordinate change followed.

The vehicle flow within the visual field of the camera can be measuredby this processing. It is possible to determine by monitoring this datathat an abnormality has occurred, for example, when the vehicle speedhas abnormally decreased.

When the end coordinate of a vehicle is extracted, the range of vehicleexistence, i.e., the coordinates at the extreme right and left ends ofthe vehicle are measured at the same time in order to monitor for anabnormal run as described below.

(2) Abnormal run monitor section

A small fallen object or the like cannot be easily detected by thefunction of stationary vehicle detection described below. In thepresence of a fallen object, however, the vehicle is normally drivenavoiding the object, and therefore an abnormal running pattern isgenerated. This function is for monitoring such an abnormal runningpattern. Many vehicles running in abnormal manner are liable to straddlethe lane boundary, frequently change lanes or change the speed suddenly.

In order to detect such phenomena as mentioned above, the presentinvention uses the processing as shown in FIG. 38. A road image isdesignated as 40.

A vehicle is extracted from an input image as in the traffic flowmeasurement.

The various coordinates of the vehicle are determined from the vehicleimage extracted. They include the distance ds to the coordinate at theextreme left end of the vehicle, the distance de to the coordinate atthe extreme right end of the vehicle and the central coordinate.

Each vehicle is followed at the time of traffic flow measurement, andthe change of the coordinates thereof is determined. As a result, theinstantaneous speed and direction of movement are obtained.

After the above-mentioned processings, the respective data are reservedas an accumulated data as shown in FIGS. 39A to 39C for abnormalitydetermination.

Accumulated data on the coordinates at the extreme left and right endsprovide information to determine a range in which vehicles are runningfrequently within a lane as shown in FIG. 39A. It is thus possible todetermine that a vehicle is running abnormally when the frequency ofdisplacement from a normal drive range exceeds a predetermined thresholdvalue.

Determination can also be made on whether the data on speed or directionof movement, if managed in similar fashion as shown in FIGS. 39B and39C, frequently assumes an abnormal value.

(3) Function of stationary vehicle detection

In the case where there is any stationary vehicle present within thevisual field of the camera, direct detection of the stationary object ismore desirable than indirect measurement like the monitoring of anabnormal run. More specifically, a stationary object detectable byimaging (a stationary vehicle, a somewhat large fallen object, etc.) isdetected directly.

A method of detecting a stationary object is shown in FIG. 40. Anoutline of this processing is described below.

An image is inputted at predetermined time intervals dt as indicated byf(t- dt) 42 and f(t) 43, and a background image 41 stored in advance isremoved from each input image.

The images 44, 45 from which the background has been removed have only avehicle image remaining therein. A moving object is detected by use ofthese images. A method of detecting a moving object is by determiningthe speed as in the measurement of traffic flow. In the case underconsideration, however, a moving object is detected by the featuresbetween image frames. More specifically, two images are subjected to adifferentiation processing to extract a moving area (46).

The area from which a moving area image 46 is removed from an imagelacking the background makes up a candidate image of a stationary object(47). Upon confirmation that the position of this object remainsstationary, it is determined that there exists a stationary object as afinal result (48).

A stationary object or a fallen object can be detected within the visualfield of the camera by executing the above-mentioned processing.

(4) Function of congestion measurement

An abnormal phenomenon outside of the visual field of the TV camera isimpossible to detect directly. In view of this, a phenomenon as arepercussion of an abnormality occurring outside of the camera whichenters the visual field thereof, i.e., the congestion degree, ismeasured. In the case where the congestion degree assumes a valuedifferent from a normal one, it may be decided that some abnormality hasoccurred in the direction forward of the vehicle.

The congestion (traffic jam) degree may be measured by various methods.The method employed in the present invention uses no traffic flow databut is based on the macroscopic measurement of the number and speed ofvehicles taken by such expressions as "vehicles are many", "vehicles arefew", "vehicle speed is high" or "vehicle speed is low" as shown in FIG.41. This is due to the fact that with the intensification of vehiclecongestion, the images taken by the TV camera are superposed one onanother, thereby making impossible the microscopic measurement of thenumber and speed of vehicles. Especially, vehicles appear superposedvery often in a tunnel where TV cameras are installed at low level.

The processing will be described briefly below.

An input image f(t) 50 is differentiated (with a profile perpendicularto the running direction extracted) (spatial differentiation 51), andthe feature amount relating to the vehicle quantity is calculated (52).In the case under consideration, a differentiated image is binary-codedand the number of vehicle profiles is determined.

Further, an image f(t+dt) 53 is inputted at predetermined time intervalsdt (say, 200 ms), and a difference image is determined for each pictureelement of f(t) 50 and f(t+dt) 53 (temporal differentiation 54). If avehicle is moving, a differentiation image in some form or otherappears. In the case where the vehicle remains stationary, on the otherhand, no information is obtained. The feature amount (such as width) ofan image obtained from the differentiation image is therefore determinedas a data relating to speed (55).

The feature amounts relating to the vehicle quantity and speed mentionedabove are applied to a decision function 56 engaged in learning inadvance for calculation of the congestion degree. A neural network isused for the decision function under consideration.

An abnormal phenomenon can be monitored by the feature amount like thevehicle quantity or speed as well as by the congestion degree determinedfrom the above-mentioned processing.

(5) Local overall determination section

The processing by the local overall determination section is executedmainly for improving the reliability of the information obtained fromthe functions of "traffic flow measurement", "measurement of abnormalrun", "detection of stationary vehicles" and "congestion degreemeasurement". Abnormal phenomena include local ones such as stationaryvehicles (accidents) and overspeed, and abnormalities covering wideareas like traffic congestion. An alarm should be issued against a localabnormality at the time point of detection by each image processingunit. Nevertheless, the problem of measurement accuracy makes itnecessary to check the measurement result. Also, the manner in which analarm is issued against a congestion apparently covering a wide rangedepends on whether the particular congestion is confined within thevisual field of the camera or covers all the measurement points. As aresult, information on an abnormality covering a wide range (congestiondegree, measurement result of traffic flow, etc.) is sent to a hostsystem to make an overall decision (determination).

The information applied to the local overall determination sectionincludes:

Traffic flow: Speed is high or low, traffic volume large or small

Abnormal run: Frequent or infrequent overspeed, frequently orinfrequently displaced from normal running course

Stationary vehicle: Presence or absence

Congestion degree: high, middle or low

Of these information, local abnormalities relate to whether a case ofoverspeed or stationary vehicle is detected or not. As to the overspeed,a contradictory data indicating an overspeed at the time of congestionmeasured by the abnormal run function is considered impossible and iscancelled. The relationship between the measurement result of thecongestion measurement function and the speed information is thuschecked and a contradictory data is cancelled to prevent an overalarm.The information obtained in detecting a stationary vehicle, however, isreliable and may be used to issue an alarm immediately upon detection.

If the degree of an error developed by the speed data measured inaccordance with the congestion level is grasped in advance, on the otherhand, the speed data can be corrected in accordance with the congestiondegree determined. This is also the case with the number of vehicles.

The local overall decision section checks for a data contradiction, andtransmits information to a central determination section 22 as a hostsystem, while at the same time controlling the information on thepresence of a stationary vehicle or an abnormal run as determined undera condition allowing accurate speed measurement ("unbusy state") in sucha manner as to permit an immediate alarm output.

(6) Central overall determination section

In the event that TV cameras are installed at intervals of 200 m, forexample, the range actually measurable by imaging covers only about onehalf of this distance. An abnormal phenomenon occurring at a positionnot monitored by imaging, therefore, is required to be subjected tointerpolation (estimation) from the result of processing of adjacent TVcameras.

An interpolation processing described below is effected according to thepresent invention.

(i) Spatial interpolation

The spatial interpolation at the central overall determination sectionis to spatially grasp and interpolate the result of processing the imageof each TV camera. The spatial interpolation enables a determination onwhether a given congestion is a chronic one or due to an accident or astationary vehicle, or an estimation of any abnormality in an area notcovered by the camera. For example, the congestion degree at each groundpoint is plotted two-dimensionally as shown in FIG. 42, and it isdecided that some abnormality has occurred at or in the vicinity of apoint where the congestion degree has considerably changed as comparedwith the neighbouring conditions. In FIG. 42, the congestion degree ofthe left lane is low, while that of the opposite lane (passing lane) ishigh. It is, therefore, estimated that abnormally many vehicles changelanes in this section.

More specifically, the congestion degree obtained from each imageprocessing unit is studied in overall fashion. It is decided that anabnormality is caused outside of the measurement section if traffic iscongested in all sections, or that the congestion is a chronic one ifthe traffic congestion is limited to part of the measurement range, sothat an abnormality decision is made only when traffic is congested onlyin part of the measurement range.

All the information acquired are processed as data, and any part wherethe congestion degree is locally different is searched for. If there isfound any such part, an abnormality decision is made. Otherwise, adecision is made that the traffic is normal. The data is analyzed eitherby a general waveform (one-dimensional data on congestion degree withrespect to ground point) analysis technique or by a method in which awaveform is directly inputted to a neural network to detect a changepoint.

Objects of data analysis may include not only the congestion degree butalso the data on abnormally running vehicles monitored (lane data, speeddata, etc.). It is possible to make an abnormality decision whenvehicles are displaced from a normal course only at a part of the range.

In general, an attendant observes the monitor screen to check for anyabnormal phenomenon. If a spatial graph as shown in FIG. 42 is indicatedto present the result of automatic decision to the monitor attendant,the conditions of the entire measurement range become easy to grasp.

An abnormality is an invisible phenomenon, and therefore the typethereof cannot be specified, although a decision as to the presence orabsence of an abnormality is possible. The coordinate distribution,speed distribution, etc. of vehicles, instead of the congestion degreeplotted in FIG. 42, may be spatially determined.

(ii) Temporal interpolation

The data measured at each image processing unit is collected and managedin time series as shown in FIG. 43. A decision is made on the time ofoccurrence or on whether the abnormalities are of primary nature (suchas the effect of a noise specific to the image processing) or ofsustaining nature. This decision is made by waveform analysis as in thecase of the spatial interpolation described above.

As a result, a noise-related abnormality is removed for an improvedsystem reliability. Also, the indication to the monitor attendantfacilitates the grasping of a chronological change as in the case ofspatial interpolation.

An example of hardware configuration for realizing the abnormalphenomenon detection mentioned above is shown in FIG. 44. Video imagesfrom a plurality of TV cameras 60 installed appropriately are applied toimage processing units 100, which execute "traffic flow measurement","monitoring of abnormally-running vehicles", "detection of stationaryvehicles" and "congestion degree measurement", and transmit the resultof the executions to a central overall determination section 22 as ahost system.

Each of the image processing units 100 includes an A/D converter 70 forconverting a video signal of the TV camera 60 into a digital signal, animage memory 71 for storing the resulting data, an image processor 72for processing the data of the image memory 71, a data output section 73for transmitting the data to the central overall determination section22, and a CPU 74 for controlling these devices, as shown in FIG. 45. Theimage processor 72 is capable of executing the inter-image operationssuch as subtraction and addition of images or a spatial sum-of theproducts operation such as differentiation, binary coding or histogramprocessing. The local overall decision is processed at the CPU 74.

The central overall determination section 22, as shown in FIG. 46,includes an interface section 74 for transmitting and receiving a datawith each image processing unit 100 not shown, a data accumulator 75 forreserving the data thus obtained, and a data analyzer 76 for subjectingthe accumulated data to spatial and time interpolation.

According to the present embodiment, as described above, variousabnormal phenomena are capable of being detected by the functions of"traffic flow measurement", "detection of abnormally-running vehicles","detection of stationary vehicles", "congestion degree measurement" and"inter-camera interpolation".

In addition to the conditions in the tunnel, as described above, theroad conditions in general can of course be monitored with equal effectaccording to the present invention.

We claim:
 1. An object recognition system for photographing an object bya TV camera and recognizing movement of the object by processing imagesfrom the TV camera, comprising:a template extracting circuit extractingpartial templates from an image of an object; a following means forfollowing the object by correlation calculation between the partialtemplates and an input image; and means for separating/integrating thepartial templates on a basis of a selected one of a history ofcoordinates followed and a knowledge relating to the object.
 2. Anobject recognition system according to claim 1, wherein said templateextracting circuit extracts images of a plurality of partial areasformed on an object-incoming side of an input image as templates.
 3. Anobject recognition system according to claim 1, wherein said templateextracting circuit extracts an input image corresponding to an areahaving selected one of a large spatial frequency feature and a largetime frequency feature as a template of predetermined size.
 4. An objectrecognition system according to claim 1, wherein the template extractingcircuit extracts an area representing a similarity of a moving objectfrom an input image and extracts an input image corresponding to aparticular area as said templates.
 5. An object recognition systemaccording to claim 1, wherein said separation/integration circuitseparates or integrates templates having a similar speed and directionof movement of an object on a basis of the trace of movement determinedfor each template.
 6. An object recognition system according to claim 1,wherein said following means comprises:a means for learning a pastbehavior pattern by a neural network; a means for predicting futurebehavior from the past behavior pattern and a present behavior pattern;and a means for determining a range of correlation calculation withrespect to templates.
 7. An object recognition system according to claim1, further comprising a template updating circuit for updating atemplate image stored by use of an input image.
 8. An object recognitionsystem according to claim 7, wherein the template updating circuitmatches patterns between an input image f(t) at time point t and atemplate g, and in a case where a coordinate P with a similarity degreelarger than a predetermined threshold value is determined, an image inthe vicinity of the coordinate P of input image f(t) is registered newlyas a gray scale template g'.
 9. An object recognition system accordingto claim 7, wherein the template updating circuit matches patternsbetween an input image f(t) at time point t and a template g, and in acase where a coordinate P with a similarity degree larger than apredetermined threshold value is determined, a new template expressed byg'=αf'(t)+βg (σ, β: constants) using an image f'(t) in the vicinity ofthe coordinate P of all the input images f(t) is registered.
 10. Anobject recognition system for photographing an object by a TV camera andrecognizing movement of the object by processing input images from theTV camera, comprising:a template extracting circuit extracting aplurality of images in a template area of predetermined size from aninput image f(t) at a time point t; a template memory circuit storing aplurality of images obtained by the template extracting circuit; apattern matching circuit determining a movement point of each templateby shade pattern matching between each template stored in the templatememory circuit and the input images at incrementing time points obtainedfrom the TV camera; a trace management table storing a relationshipbetween a time point and a movement point for each gray scale templateobtained at the gray scale pattern matching circuit; and aseparation/integration circuit deciding which templates represent animage of a same object by use of the trace management table.
 11. Asystem for measuring an intersection traffic flow comprising:a templateextracting circuit extracting a plurality of images of a template areaof predetermined size from an image at time point t; a gray scaletemplate memory circuit storing a plurality of templates obtained withthe template extracting circuit; a pattern matching circuit matchingpatterns between each of the templates stored in said template memorycircuit and input images at time points (t+dt), (t+2dt), . . . (t+ndt)(n: integral number) obtained from a TV camera and determining amovement point of each of said templates; a template updating circuitsequentially updating the templates of said template memory circuitusing an input image; a trace management table storing a relationshipbetween a time point and a moving point for each template obtained withthe pattern matching circuit; a separation/integration circuit decidingon which templates represent a same object by use of the tracemanagement table; and a traffic parameter measurement section measuringa number and speed of vehicles passed, a number of vehicles turned to aright/left by following a vehicle position obtained.
 12. A vehiclerecognition system for photographing a road by a TV camera andrecognizing movement of vehicles by processing images from the TVcamera, comprising:a template memory circuit for storing several typesof vehicles images as images in advance; and a pattern matching circuitfor determining a vehicle position within an input image by patternmatching between the templates stored in the template memory circuit andthe input image obtained from the TV camera.
 13. A vehicle recognitionsystem according to claim 12, further comprising a template patternpreparation circuit having a plurality of template patterns T(0) to T(n)for determining a similarity degree between the template patterns andfor preparing a new template pattern from patterns similar to eachother.
 14. A vehicle recognition system for photographing a road by a TVcamera and recognizing vehicle movement by processing images from the TVcamera, comprising:a template extracting circuit for extracting an imagepattern corresponding to a vehicle as a template from an input imageobtained by the TV camera; a template memory circuit for storing atemplate obtained with the template extracting circuit; a patternmatching circuit for determining a vehicle position within a subsequentinput image by pattern matching between the template stored in thetemplate memory circuit and said subsequent input image; and a templateextracting circuit for cutting out an area of predetermined size from animage entered by a vehicle, determining an extracting position in such amanner as to locate the vehicle at a center thereof from features of anextracted image, and registering an input image of the extractingposition and a vicinity thereof as templates.
 15. A traffic flowmeasurement system for photographing a road by a TV camera andrecognizing vehicle movement by processing images from the TV camera,comprising:a template memory circuit storing a vehicle imagecorresponding to a vehicle as a template a pattern matching circuitpattern matching between the template stored in the template memorycircuit and a subsequent input image obtained from the TV camera; and avehicle speed and direction determining means for determining a vehicleposition by pattern matching with respect to the input image at a giventime point, further determining the vehicle position with respect to theinput image at a given time point and determining at least a speed anddirection of movement of vehicles, in accordance with a change incoordinates of determined positions; wherein said vehicle speed anddirection determining means determines a range of pattern matching ofthe template against the input image f(t+Δt) at a time point t+Δt on abasis of the speed and direction of movement obtained from a result ofpattern matching of images f(t-Δt), f(t) at time points (t-Δt) and trespectively.
 16. A pattern matching system for executing patternmatching between a template and an input image, comprising:a templatememory circuit storing a pattern of an image; a pattern matching circuitpattern matching between a template stored in the template memorycircuit and a subsequent input image obtained from a TV camera; and atemplate updating circuit sequentially updating the templates of thetemplate memory circuit using input images; wherein pattern matching iseffected with a given template g against an input image f(t) at timepoint t, and in a case where a coordinate P is determined to have asimilarity degree obtained by pattern matching larger than apredetermined threshold value, an input image f(t) of the coordinate Pand a vicinity thereof is registered as a new template g'.
 17. A patternmatching system for executing pattern matching between a template and aninput image, comprising:a template memory circuit storing a pattern ofan image; a pattern matching circuit pattern matching between a templatestored in the template memory circuit and a subsequent input imageobtained from a TV camera; and a template updating circuit sequentiallyupdating the templates of the template memory circuit using inputimages; wherein pattern matching is effected with an input image f(t) ata time point t against a template g, and in a case where a coordinate Pis determined to have a similarity degree larger than a predeterminedthreshold value, an input image f(t) of the coordinate P and a vicinitythereof is used to register a template expressed as g'=αf'(t)+βg (α, β:constants) as a new template.
 18. A pattern matching system forexecuting pattern matching between a template and an input image,comprising:a template memory circuit storing a pattern of an image; apattern matching circuit pattern matching between a template stored inthe template memory circuit and a subsequent input image obtained from aTV camera; and a template updating circuit sequentially updating thetemplates of the template memory circuit using input images; wherein atemplate is updated while changing a template size in accordance with asize of an object.
 19. A system for following, in a picked-up image, amoving object of a shape and size changing with the direction ofmovement and distance from the image pick-up means, comprising:atemplate memory circuit for storing a pattern of an image of the movingobject as a template; a pattern matching circuit for determining aposition of the moving object by pattern matching between the templatestored in the template memory circuit and an input image obtained fromthe image pick-up means; and a template updating circuit for preparingan updated template for a next pattern matching from the template of themoving object cut out of a present input image and a preceding templateof the moving object and storing the updated template in the templatememory circuit.
 20. An abnormality detection system for detecting anabnormal phenomenon by photographing a road with a plurality of TVcameras installed at predetermined intervals on the road and processingimages from the TV cameras, comprising:a plurality of image processingunits including a local overall decision section having functions ofprocessing images from the TV cameras and measuring a speed and numberof vehicles, monitoring abnormally-running vehicles, detecting astationary object and measuring a congestion degree, and correcting dataon the speed and number of vehicles and the abnormally-running vehiclemonitor functions in accordance with the congestion degree; and acentral overall decision section making an overall decision by spatialand temporal interpolation of a result of measurement at the imageprocessing units.
 21. An abnormality detection system according to claim20, wherein the central overall decision section includes a means forcollecting several types of information from a plurality of imageprocessing units, a spatial interpolation means for grasping saidinformation spatially and making an abnormality decision only when saidinformation undergoes a local change, and a time interpolation means forgrasping a plurality of information in time series and deciding on aninitial time of abnormality occurrence and an instantaneous abnormality.22. An abnormality detection system for detecting an abnormal phenomenonby photographing a road with a TV camera and by processing images fromthe TV camera, comprising:a local overall decision section havingfunctions of measuring a speed and number of vehicles, monitoringabnormally-running vehicles, detecting a stationary object and measuringa congestion degree for correcting data on the speed and number ofvehicles and abnormally-running vehicle monitor functions in accordancewith the congestion degree; wherein the function of monitoringabnormally-running vehicles is performed using a means for extractingvehicles from an input image, a means for determining an extreme leftcoordinate, extreme right coordinate and end coordinate of vehicles froma vehicle image extracted, a means for calculating an amount of vehiclemovement from coordinates thus determined and an image inputted at anext time point, a means for accumulating resulting coordinate valuesand speed and direction of movement for a number of vehicles involved,and means for making an abnormality decision when coordinates exceed athreshold value of a frequency of vehicles running outside apredetermined position, a frequency of vehicles running at other than apredetermined speed and a frequency of vehicles moving in other than apredetermined direction, respectively.
 23. An abnormality detectionsystem for detecting an abnormal phenomenon by photographing a road witha TV camera and by processing images from the TV camera, comprising:alocal overall decision section having functions of measuring a speed andnumber of vehicles, monitoring abnormally-running vehicles, detecting astationary object and measuring a congestion degree for correcting dataon the speed and number of vehicles and abnormally-running vehiclemonitor functions in accordance with the congestion degree; wherein thelocal overall decision section cancels a speed data obtained with a highcongestion degree, when the speed data is higher than a predeterminedthreshold value.
 24. An abnormality detection system for detecting anabnormal phenomenon by photographing a road with a TV camera and byprocessing images from the TV camera, comprising:a local overalldecision section having functions of measuring a speed and number ofvehicles, monitoring abnormally-running vehicles, detecting a stationaryobject and measuring a congestion degree for correcting data on thespeed and number of vehicles and abnormally-running vehicle monitorfunctions in accordance with the congestion degree; wherein the localoverall decision section corrects data on the speed and the number ofvehicles determined, in accordance with the congestion degree.